Method and apparatus for providing emotion-adaptive user interface

ABSTRACT

The present invention relates to a method and apparatus for providing an emotion-adaptive user interface (UI) on the basis of an affective computing service, in which the provided service is configured with at least one of a service operation condition, a service end condition, and an emotion-adaptive UI service type on the basis of purpose information of the service, and the detailed pattern is changed and provided on the basis of the purpose information and the usefulness information of the service.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims priority to Korean Patent Application No.10-2019-0005622, filed Jan. 16, 2019, the entire contents of which isincorporated herein for all purposes by this reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a method and apparatus for providing anemotion-adaptive user interface (UI) capable of being utilized in anaffective computing service based on user emotion information.

Description of the Related Art

Recently, research and development related to life-logging for storingand analyzing daily experience data and providing useful services tousers has steadily progressed. For example, there are various personalwearable and portable devices, and such a device may sense bio-signalssuch as user's physical activity and pulse and thus provide an operationmode, for example vibration according to whether a user reaches a targetphysical activity during a day. In recent years, there has been a demandfor an affective computing service that is capable of actively improvingthe quality of user's physical and mental health on the basis of theuser's experience data, which goes beyond a simple function of providinga behavior monitoring or a notification service to a user.

Service and user interface (UI) in terms of affective computing aim atimproving service quality with respect to usefulness, usability, andemotion. Accordingly, in order to provide an affective computingservice, it is necessary to provide an emotion-adaptive user interface(UI) that efficiently recognizes user emotion information and operatesto conform to the purpose of an affective computing service.

A document 1 as the related art (Apparatus and Method for EmotionInteraction Based on Biological Signals, US2014-0234815) is related to atechnology that recognizes a user's emotion state based on a bio-signaland transfers an emotion between the person and the product.

A document 2 (Method for Emotion Communication between Emotion SignalSensing Device and Emotion Service Providing Device, U.S. Pat. No.8,775,186) discloses an emotion communication protocol between anemotion signal sensing device and an emotion service providing device.

However, there is insufficient research on a method of dynamicallychanging and providing the UI pattern in a manner to satisfy the servicepurpose on the basis of the purpose information of an affectivecomputing service, the user's emotion recognized in real time, and theuser experience (UX) information about the existing UI.

SUMMARY OF THE INVENTION

The objective of the present invention is to provide an emotion-adaptiveUI for an affective computing service, the emotion-adaptive UI beingcapable of actively improving quality of user's physical and mentalhealth on the basis of user experience data.

The present invention is intended to provide a service for which UI issuited to the purpose thereof.

The present invention is intended to utilize user experience information(UX) so as to recognize the service purpose and to be operated inaccordance with the user's emotion.

The present invention is intended to provide service based on useremotion information recognized on the basis of bio-signals andvoice/video data.

The present invention is intended to provide an emotion-adaptive UIbased on the usefulness information of the corresponding UI dynamicallycalculated on the basis of the emotion service purpose information andthe user emotion information.

It is to be understood that technical problems to be solved by thepresent disclosure are not limited to the aforementioned technicalproblems and other technical problems which are not mentioned will beapparent from the following description to a person with an ordinaryskill in the art to which the present disclosure pertains.

The present invention relates to a method and apparatus for providing anemotion-adaptive UI on the basis of an affective computing service.

The present invention may include an emotion recognition unit acquiringan input signal for emotion information of a user, an emotion-adaptiveUI execution unit providing a service based on the acquired input signalfor the emotion information through the emotion-adaptive UI andcalculating usefulness information for the service, and anemotion-adaptive UI/UX management unit reflecting a feedback based onthe usefulness information for the provided service. The service may beconfigured with at least one of a service operation condition, a serviceend condition, and an emotion-adaptive UI service type on the basis ofpurpose information of the service, and the emotion-adaptive UI servicetype is configured with one or more detailed patterns.

According to an embodiment of the present invention, the service isprovided, the detailed pattern may be changed and provided on the basisof at least one of the purpose information and the usefulnessinformation included in user experience information.

According to an embodiment of the present invention, the apparatus maybe provided to operate one or more emotion-adaptive UI service typesfrom sound service, display service, vibration service, and tactileservice.

According to an embodiment of the present invention, the apparatus maycorrespond to a wearable device.

According to an embodiment of the present invention, the usefulnessinformation may correspond to an evaluation index for the serviceprovided through the emotion-adaptive UI, and may be determined on thebasis of the purpose information of the service.

According to an embodiment of the present invention, when the service isprovided, the detailed patterns included in the service may be changedin a manner to satisfy the purpose information of the service on thebasis of the purpose information of the service.

According to an embodiment of the present invention, in a case that thedetailed pattern is changed in a manner to satisfy the purposeinformation of the service, when the usefulness calculated for thedetailed pattern is equal to or greater than a threshold value accordingto a result of providing the service having the detailed pattern, it maybe determined that the purpose information of the service is satisfied.

According to an embodiment of the present invention, the serviceusefulness information of the usefulness information may be calculatedon the basis of at least one of the emotion information at the start ofthe detailed pattern of the service, the emotion information at the endof the detailed pattern, and the purpose information of the service.

According to an embodiment of the present invention, the detailedpattern usefulness information of the usefulness information may becalculated on the basis of at least one of a difference between theemotion information at the start of the detailed pattern and the emotioninformation at the end of the detailed pattern and the purposeinformation of the service.

According to an embodiment of the present invention, the serviceoperation condition and the service end condition may include at leastone of time attribute for operating the service, current emotioninformation, target emotion information, and place information.

According to an embodiment of the present invention, theemotion-adaptive UI service type may include a sound service, a displayservice, a vibration service, and a tactile service, and the serviceprovided through the emotion-adaptive UI may include at least one of asound service, a display service, a vibration service, and a tactileservice.

According to an embodiment of the present invention, the detailedpattern may correspond to the detailed attribute for theemotion-adaptive UI service type, and correspond to a method ofcontrolling the emotion-adaptive UI service type.

According to an embodiment of the present invention, when the service isprovided, the purpose information of the user experience information forthe service may be checked, and the detailed pattern included in theservice may be determined on the basis of the usefulness informationincluded in the corresponding information.

According to an embodiment of the present invention, the input signalfor the user emotion information may include at least one of bio-signaldata, voice data, and video data of the user, and the emotioninformation may be recognized on the basis of the input signal using anemotion recognition model.

According to an embodiment of the present invention, when defaultpurpose information is selected as the purpose information of theservice, the default purpose information may include emotion awakeningpurpose, emotion control purpose, and emotion consolation purpose.

According to an embodiment of the present invention, when a feedback isreflected on the operation of the emotion-adaptive UI based on theusefulness information for the provided service, information on thedetailed pattern for the provided service may be updated.

According to the present invention, it is possible to provide anemotion-adaptive UI for affective computing services that are capable ofactively improving quality of user's physical and mental health on thebasis of user experience data.

The present invention has an advantage that provides a service for whichUI is suited to the purpose thereof.

The present invention has an advantage that utilizes the user experienceinformation (UX) so as to recognize the service purpose and to beoperated in accordance with the user's emotion.

The present invention has an advantage that a service based on useremotion information recognized on the basis of bio-signals andvoice/video data can be provided.

The present invention has an advantage that a an emotion-adaptive UIbased on the usefulness information of the corresponding UI dynamicallycalculated on the basis of the purpose information of the emotionservice and the user emotion information can be provided.

Through the method of providing the emotion-adaptive UI according to thepresent invention, there is an effect that a service capable of adaptingto a variance of user's emotion can be provided.

According to the present invention, it is possible to provide a UI thatis personalized to be suitable for user's physical and emotionalsituations by utilizing the user experience information.

Through the implementation of the present invention, a UI suitable forthe purpose of various affective computing services can be operated.

Effects that may be obtained from the present disclosure will not belimited to only the above described effects. In addition, other effectswhich are not described herein will become apparent to those skilled inthe art from the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features, and other advantages of thepresent invention will be more clearly understood from the followingdetailed description when taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a flowchart illustrating a method of providing anemotion-adaptive user interface (UI) on the basis of affective computingaccording to an embodiment of the present invention;

FIG. 2 is a diagram illustrating a configuration of a service system inwhich the service is operated according to an embodiment of the presentinvention;

FIG. 3 is a diagram illustrating an attribute configuration of anemotion-adaptive UI according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating a functional block structure of theemotion-adaptive UI performed in a wearable device or a portable deviceaccording to the present invention;

FIG. 5 is a diagram illustrating operations of an emotion-adaptive UIpattern according to an embodiment of the present invention;

FIG. 6 is a diagram illustrating calculation of usefulness informationon emotion variance to calculate the usefulness of an emotion-adaptiveUI and detailed patterns;

FIG. 7 is a block diagram illustrating an emotion-adaptive UI executionunit 320 according to the present invention;

FIG. 8 is a diagram illustrating an embodiment of a wrist wearabledevice 110 that supports executing the emotion-adaptive UI according tothe present invention; and

FIG. 9 is an operation flowchart illustrating an emotion-adaptive UIaccording to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Hereinbelow, exemplary embodiments of the present disclosure will bedescribed in detail with reference to the accompanying drawings suchthat the disclosure can be easily embodied by one of ordinary skill inthe art to which this disclosure belongs. However, the presentdisclosure is not limited to the exemplary embodiments.

In describing embodiments of the present disclosure, it is noted thatwhen the detailed description of known configurations or functionsrelated to the present disclosure may make the gist of the presentdisclosure unclear, the detailed description thereof will be omitted.Also, portions that are not related to the present disclosure areomitted in the drawings, and like reference numerals designate likeelements.

In the present disclosure, when an element is “coupled to”, “combinedwith”, or “connected to” another element, it can be directly coupled tothe other element or intervening elements may be present therebetween.Also, when a constituent “comprises” or “includes” an element, unlessspecifically described, the constituent does not exclude other elementsbut may further include the elements.

In the present disclosure, the terms “first”, “second”, etc. are onlyused to distinguish one element from another element. Unlessspecifically stated otherwise, the terms do not denote an order orimportance. Thus, without departing from the scope of the presentdisclosure, a first element of an embodiment could be termed a secondelement of another embodiment. Similarly, a second element of anembodiment could also be termed a first element of another embodiment.

In the present disclosure, elements that are distinguished from eachother to clearly describe each feature do not necessarily denote thatthe elements are separated. That is, a plurality of elements may beintegrated into one hardware or software unit, or one element may bedistributed into a plurality of hardware or software units. Accordingly,even if not mentioned, the integrated or distributed embodiments areincluded in the scope of the present disclosure.

In the present disclosure, elements described in various embodiments donot denote essential elements, and some of the elements may be optional.Accordingly, an embodiment that includes a subset of elements describedin another embodiment is included in the scope of the presentdisclosure. Also, an embodiment that includes the elements which aredescribed in the various embodiments and additional other elements isincluded in the scope of the present disclosure.

Hereinafter, the embodiments of the present disclosure will be describedwith reference to the accompanying drawings.

The present invention relates to a method and apparatus for providing anemotion-adaptive UI for an affective computing service. Morespecifically, the present invention relates to a method of providing aUI that adapts to the user's emotion and an apparatus therefor, in whichusefulness information dynamically calculated in a service process forthe UI is extracted on the basis of recognized emotion information, andthe existing user experience UX information including the correspondingusefulness information for the corresponding UI is utilized.

In various affective computing services that actively improve thequality of user's physical and mental health, the emotion-adaptive UIshould operate according to the purpose intended to be ultimatelyprovided to the user by the service.

The emotion-adaptive UI according to the present invention may operatein accordance with the purpose of various emotion services. In thepresent invention, the emotion-adaptive UI may be also referred to asthe emotion-adaptive UI.

For the operation of the present invention, detailed patterns of theemotion-adaptive UI may be dynamically adjusted on the basis of useremotion information, emotion service purpose information, and userexperience (UX) information. Herein, the user emotion information maycorrespond to the information recognized on the basis of bio-signal,voice, and video data of the user. The user experience information mayinclude user's usefulness information on the emotion-adaptive UI.Herein, the user's usefulness information included in the userexperience information may correspond to the evaluation index for theemotion-adaptive UI when the emotion-adaptive UI was operatedpreviously.

Emotion information recognized from user experience data includes aspecific emotion label such as sadness, joy, and anger, and varioussensitivity labels such as neutral, awake, relaxed, and extremelyrelaxed (e.g. asleep), positive state, and negative state, describingthe user's biorhythm and emotions.

The purpose of the various emotion services according to the presentinvention may include emotion awakening, emotion control, emotionconsolation, and the like.

The purpose of the emotion awakening is to change the detailed patternof the emotion-adaptive UI according to the periodically recognizedbiorhythm state and emotion information, thereby helping the user reachthe awake.

The purpose of emotion control is to help a user to properly recognizeand control his/her emotions for a user-specific emotion state.

The purpose of emotion consolation is to provide the emotional comfortto the user by changing the detailed pattern of the emotion-adaptive UIaccording to the user's bio-signals and emotions.

The emotion-adaptive UI is operated in such a manner to change thepattern of the emotion-adaptive UI on the basis of the purposeinformation of such an emotion service.

However, the present invention is not limited to the above-mentionedpurposes, and the above purposes may correspond to the purposeinformation that may be fundamentally set in the present apparatus.

Also, the user may set the purpose information according to anembodiment of the present invention.

The emotion-adaptive UI is operated in such a manner to dynamicallychange the emotion-adaptive UI pattern in a manner to satisfy thepurpose of the service.

In addition, the emotion-adaptive UI of the present invention extractsthe usefulness information of the corresponding UI dynamicallycalculated in the service process on the basis of the service purposeinformation and the recognized emotion information, and uses the userexperience information on the extracted usefulness information to changethe pattern of the emotion-adaptive UI.

Hereinafter, embodiments of the present invention will be described withreference to the accompanying drawings. The present invention will bedescribed in detail with reference to the portions necessary forunderstanding the operation according to the present invention.

FIG. 1 is a flowchart illustrating a method of providing anemotion-adaptive user interface (UI) based on affective computingaccording to an embodiment of the present invention.

Although there are many technologies aimed at providing (recommending) asuitable service according to emotion in the related art, a method ofreaching the service purpose with maximum efficiency is disclosedaccording to the present invention. In this case, the service purposemay be classified into a default service purpose set fundamentally and apurpose defined by a user.

Herein, the present invention adjusts the detailed pattern of theregistered emotion-adaptive UI, in which the detailed pattern isadjusted by calculating the usefulness according to variance information(experience) of the user's emotion state.

In addition, the present invention may extract reaction information fromthe emotion-adaptive UI. Herein, the extracted reaction information isstored as experience information of the emotion-adaptive UI, and thusreferenced when setting the initial value for the next operation of thecorresponding emotion-adaptive UI.

The apparatus for providing the emotion-adaptive UI may first obtain aninput signal for the user emotion information (S110).

Herein, the input signal to the user emotion information may include atleast one of bio-signal data, voice data, and video data of the user.Also, the emotion information may be recognized on the basis of theinput signal using an emotion recognition model according to anembodiment of the present invention.

The apparatus may then provide the service based on the emotioninformation through the emotion-adaptive UI (S120).

According to the present invention, a service provided through theemotion-adaptive UI includes at least one of a service operationcondition, a service end condition, and an emotion-adaptive UI servicetype on the basis of the service purpose information.

Herein, the service operation condition and the service end conditioninclude at least one of a time information for operating the service,current emotion information, target emotion information, and placeinformation.

According to the present invention, the emotion-adaptive UI service typemay include a sound service, a display service, a vibration service, anda tactile service. However, the present invention is not limitedthereto. Herein, the service provided through the emotion-adaptive UImay be provided with at least one of a sound service, a display service,a vibration service, and a tactile service.

According to the present invention, the emotion-adaptive UI service typemay be configured with one or more detailed patterns. Herein, thedetailed pattern may correspond to the detailed attribute for theemotion-adaptive UI service type. The detailed pattern may alsocorrespond to a method of controlling the emotion-adaptive UI servicetype.

When the service is provided according to an embodiment of the presentinvention, the detailed pattern may be changed and provided on the basisof at least one of user's experience information including the purposeinformation and usefulness information of the service.

When the service is provided according to an embodiment of the presentinvention, the detailed patterns included in the service may be changedin a manner to satisfy the service purpose information, on the basis ofthe service purpose information.

The provided service is provided so that the detailed pattern thereofmay be modified in the direction that satisfies the service purposeinformation. In this case, according to the result of providing theservice having the detailed pattern, when the usefulness calculated forthe detailed pattern is equal to or larger than the threshold value, itmay be determined that the service purpose information is satisfied.Herein, the threshold value may correspond to the designated value ofthe user.

When the service is provided according to an embodiment of the presentinvention, information corresponding to the service purpose informationamong the user experience information is checked, and the detailedpattern included in the service may be determined on the basis of theusefulness information included in the corresponding information.

The apparatus may then calculate the usefulness information for theservice (S130).

According to the present invention, the usefulness information maycorrespond to the evaluation index for the service provided through theemotion-adaptive UI and mean information that is determined on the basisof the service purpose information. Herein, the usefulness informationmay include service usefulness information and detailed patternusefulness information.

The service usefulness information of the usefulness information may becalculated on the basis of at least one of the emotion information atthe start of the detailed pattern of the service, the emotioninformation at the end of the detailed pattern of the service, and theservice purpose information.

The detailed pattern usefulness information of the usefulnessinformation may be calculated on the basis of at least one of differencebetween the emotion information at the start of the detailed pattern andthe emotion information at the end of the detailed pattern, and theservice purpose information.

Finally, the apparatus may perform feedback based on usefulnessinformation for the provided services, and may reflect the feedback tothe operation of the emotion-adaptive UI (S140).

When the feedback is reflected in the operation of the emotion-adaptiveUI based on the usefulness information is reflected on the providedservice according to an embodiment of the present invention, theinformation on the detailed pattern for the provided service may beupdated.

Specific details of the operation and configuration of the presentinvention will be described referring to the following drawings.

FIG. 2 is a diagram illustrating a configuration of a service system inwhich the service is operated according to an embodiment of the presentinvention.

The service system in which the service operates according to anembodiment of the present invention may include a user-wearable device110, a user portable device 120, and an Internet of Things peripheral130.

Herein, the bio-signal and audio/video data of the user 100 used forrecognizing the user's emotion may be collected through theuser-wearable device 110, the user portable device 120, and the Internetof Things peripheral 130.

The user experience data and recognized emotion information collectedfrom these devices may be exchanged through the inter-devicecommunication method. An emotion-adaptive UI module that operates on thebasis of the exchanged user experience data and recognized emotioninformation may also be implemented in each apparatus.

The collected bio-signals may include data such as heart rate (HR),heart rate variation (HRV) calculated through the heart rateinformation, galvanic skin response (GSR), photoplethysmogram (PPG), andthe like, but the present invention is not limited thereto.

Operations (start and end) of the user-wearable device 110, the userportable device 120, and the Internet of Things peripheral 130 may bedetermined on the basis of the emotion information recognized throughthe user experience data.

In more detail, the emotion information may be recognized through theuser experience data collected in the user-wearable device 110, the userportable device 120, and the Internet of Things peripheral 130. Inaddition, the start and end of the emotion-adaptive UI operation statein which the emotion-adaptive UI operates may be determined through acombination of other attribute such as the time for the emotion-adaptiveUI operation state. In addition, the emotion-adaptive UI registered bythe user may be executed.

Herein, the emotion-adaptive UI operation may be terminated when theemotion-adaptive UI operation state has reached the designated time orthe variance in the emotion state, or an explicit end interface occursfrom the designated user through the user-wearable device or theportable device.

FIG. 3 shows an attribute configuration of an emotion-adaptive UIaccording to an embodiment of the present invention.

The service of the present invention may include at least one of aservice operation condition, a service end condition, and anemotion-adaptive UI service type on the basis of the service purposeinformation.

According to an embodiment of the present invention, the serviceoperation condition and the service end condition may correspond toattributes implemented in FIG. 3, and a combination of the attributesimplemented in FIG. 3 may be specified. However, the attributesimplemented in FIG. 3 may correspond to an embodiment of the presentinvention.

In the present invention, an attribute may correspond to a property orattribute of the service, and correspond to elements constituting theservice. Further, the attribute may correspond to the information in thepresent invention. For example, the time attribute may correspond totime information, and the emotion attribute may correspond to emotioninformation.

The service purpose information according to the present invention maycorrespond to the emotion-adaptive UI purpose 230 of FIG. 3. In thepresent invention, the service purpose information may correspond to theinformation reflecting the purpose of providing the service to the user.In the present invention, the service purpose information may correspondto a word having the same meaning as the service purpose or theemotion-adaptive UI purpose. According to one embodiment of the presentinvention, the service purpose information may correspond to informationreflecting the purpose of emotional awakening, emotional control, andconsolation.

The emotion-adaptive UI service type of the present invention maycorrespond to the emotion-adaptive UI type 240 of FIG. 3. In the presentinvention, emotion-adaptive UI type, the emotion-adaptive UI servicetype may correspond to a word having the same meaning asemotion-adaptive UI type the emotion-adaptive UI, emotion-adaptive UItype attribute or the feedback type attribute.

In order to provide the UI according to the present invention, the usermay set the service purpose information 230, the service operationcondition, the end condition of the service, and the emotion-adaptive UIservice type 240 according to an embodiment of the present invention. Inaddition, according to an embodiment of the present invention, the usermay use the services preset for service purpose information, serviceoperation condition, service end condition, and emotion-adaptive UIservice type.

A combination of several or more attributes may be represented in theservice operation condition and the service end condition.

According to an embodiment of the present invention, the serviceoperation conditions for operating the service correspond to conditionsset on the basis of a time information 210 for operating the service,current emotion information and target emotion information 220, placeinformation, and the like.

According to an embodiment of the present invention, the service endconditions correspond to conditions set on the basis of the timeinformation 210 for operating the service, the current emotioninformation and target emotion information 220, and the placeinformation.

Therefore, the attributes of FIG. 3 may correspond to an example of thepresent invention. When the emotion-adaptive UI of the present inventionis implemented in the system, types and values of attributes of the UIthat may be designated to be suitable for each service are not limited.In addition, a variety of UIs capable of being specified to suit eachservice may be set.

As the time information 210 of the emotion-adaptive UI operationalstate, a range of absolute time based on the world standard time or asemantic attribute, such as “morning”, “lunch”, or “evening” may bespecified.

The emotion information 220 of the emotion-adaptive UI operation statemay be a specific emotion label, such as sadness, joy, and anger. Inaddition, as the emotion information 220, various sensitivity labelssuch as neutral, awake, relaxed, and extremely relaxed (e.g. asleep),positive, and negative, describing the user's biorhythm and emotion maybe specified.

Table 1 shows kinds of purposes, types and detail pattern attributes ofthe emotion-adaptive UI.

According to an embodiment of the present invention, as shown in Table 1below, as the attribute of the emotion-adaptive UI purpose 230, thepurpose such as awakening, emotion control, and emotion consolation maybe specified.

The optimal UI type pattern attribute according to the user's emotionstate is determined and operated in a manner to reach the purposedesignated according to the attribute of the specified emotion-adaptiveUI purpose 230.

TABLE 1 Emotion- Emotion- adaptive UI adaptive UI Detailed patternpurpose attribute type attribute attribute Awakening Sound Type ofsound, Emotion Sound interval control Display Color, Text, Icon EmotionVibration Type of vibration, consolation Vibration interval TactileUp/down/left/right/ other tactile pattern, Intensity

The attribute of the emotion-adaptive UI purpose 230 may be used as anattribute for determining a change in the detailed pattern of theemotion-adaptive UI that is operated and, more specifically used asinformation for calculating the usefulness of the emotion-adaptive UIand the detailed patterns in order to determine a change in the detailedpattern when executing the emotion-adaptive UI.

The emotion-adaptive UI of the present invention may be operated so asto be varied according to the direction and emotion state that issuitable for the purpose attribute specified in the correspondingemotion-adaptive UI through the calculated usefulness information.

In the case where the default purpose information is selected as thepurpose information 230 of the service according to an embodiment of thepresent invention, the default purpose information includes an emotionawakening purpose, an emotion control purpose, and emotion consolationpurpose.

When registering a UI for three default service purposes, the detailedpatterns that fundamentally operate may be set automatically at thebeginning. Also, when registering the UI for the default servicepurposes, the direction in which the detailed pattern should be adjustedmay be automatically set at the beginning.

According to an embodiment of the present invention, a situation may beassumed in which an emotion service having emotion control purpose isprovided in a user terminal. From the provided service, the emotioncontrol purpose may be automatically set as the purpose of theemotion-adaptive UI according to the present invention. In addition, anoperation mode such as vibration, tactile, and sound may be arbitrarilyset by the user through the UI. When the corresponding service isactivated, the user is provided with an emotion-adaptive UI that variesaccording to his/her body rhythm, emotion state, and emotion intensity,so that the user may accurately recognize his/her emotion state and thustry to control the emotion state.

As an embodiment according to the present invention, an emotion-adaptiveUI that operates with the emotion consolation purpose may be assumed.This case may operate differently from the UI pattern for emotioncontrol purpose, which provides a linear UI pattern for the recognizedemotion state and emotion intensity. In the case of the emotion-adaptiveUI that operates with the emotion consolation purpose as an embodiment,the UI pattern may be varied to provide psychological stability to theuser when the user's negative emotion is recognized.

In addition, new purpose information may also be defined by the user forthe service purpose information 230. Herein, for the service purposeinformation 230 set by the user, the user may generate the serviceprovided by selecting at least one of service operation condition,service end condition, and emotion-adaptive UI service type.

Herein, the user may specify the start conditions of the UI having theservice purpose defined by the user, for example, the start emotioninformation 220 and the start time information 210, and the like.

Also, the user may set the end condition of the emotion-adaptive UI,such as, for example, user emotion information 220, end time information210, etc. at the time of end.

Also, the user may set the emotion-adaptive UI type 240, for example,operation mode of the device, such as vibration and sound. Accordingly,the emotion-adaptive UI may be operated in a manner to be specified forthe service purpose specified by the user.

When measuring the usefulness for the user registration service UI, thedesired service target vector of the required service purpose may bedetermined on the basis of the emotion attribute value of the registeredstart condition and the emotion attribute value of the end condition.

When the user designates the purpose information of the emotion-adaptiveUI, according to an embodiment of the present invention, the user mayregister the vibration emotion-adaptive UI to alleviate the emotion forthe environment in which a specific fear is felt.

More specifically, according to an embodiment of the present invention,a company may be selected as a “place attribute” of the operation startconditions of the emotion-adaptive UI may be selected. In addition, fearmay be selected as an “emotion attribute” of the operation startconditions of the emotion-adaptive UI.

As an “emotion attribute” of the end conditions of the emotion-adaptiveUI, an awakening degree of a low state is designated. In addition, the“purpose” of the corresponding service is set to alleviate the fear.

The emotion-adaptive UI type 240 includes sound, display, vibration,tactile, etc., each having detailed pattern attributes. However, thepresent invention is not limited thereto.

According to an embodiment of the present invention, the serviceprovided through the emotion-adaptive UI may include at least one of asound service, a display service, a vibration service, and a tactileservice. In other words, a combination of one or more types (forexample, vibration and tactile) may be provided as the emotion-adaptiveUI type according to the conditions set for the service. Also, only onetype may be provided as the emotion-adaptive UI type, depending on theconditions set for the service.

In this case, the detailed pattern corresponds to the detailed attributefor the emotion-adaptive UI service type 240. The detail pattern alsocorresponds to a method of controlling the emotion-adaptive UI servicetype.

In the present invention, the emotion-adaptive UI attribute may beexplicitly registered by the user according to an embodiment of thepresent invention. The emotion-adaptive UI attributes may also bedetermined automatically in the processes of various affective computingservices.

Each attribute element of the emotion-adaptive UI operation state may bespecified in the form of a conditional combination (AND, OR, etc.) whenall the conditions specified in the attribute are met or only a specificcondition is met. That is, according to an embodiment of the presentinvention, conditions are specified for each of the time information andthe emotion attribute, and the specified conditions are set so that theconditional combination is satisfied.

First, when the emotion-adaptive UI is explicitly designated by theuser, the user may designate the occurrence time information 210 and theuser emotion information 220 of the emotion-adaptive UI operation statethrough the user-wearable device and the portable device.

As an example of setting the attributes of the emotion-adaptive UIoperation state according to an embodiment of the present invention,when a user wants to register the emotion-adaptive UI having the purposeof reaching his/her emotion state at the sensibility of the “awakeningstate” in the morning, the user may set the emotion-adaptive UIoperation state as emotion state of “extremely relaxed state” at thetime of “7 a.m.”. That is, the user may set “awakening state” as theemotion-adaptive UI purpose 230, “7 a.m.” as the time information 210,and “extremely relaxed state” as the emotion information 220. Also, theuser may set “awakening state” as the emotion-adaptive UI purpose 230and “vibration” as the emotion-adaptive UI type 240.

Also, when the emotion-adaptive UI of the present invention isimplemented, emotion-adaptive UI operation state and theemotion-adaptive UI attribute may be automatically set at the processesof various affective computing services.

An emotion-adaptive UI provided during the user's driving of the vehiclemay be applied according to an embodiment of the present invention. Thiscase may require a user agreement on providing the emotion-adaptive UIin the navigation service during the driving. When the “extremelyrelaxed state” is detected as the emotion information 220 during theuser's driving of the vehicle, the emotion-adaptive UI for changing theuser's emotion state to the “awakening state” is provided. Herein, theemotion-adaptive UI purpose 230 may correspond to the “awakening”. Inaddition, the feedback type attribute 240 may be automatically assigned“sound”, “vibration”, “tactile”, or the like.

According to an embodiment of the present invention, some of theemotion-adaptive UI attributes may be designated automatically throughinterworking with the service, and some may be designated by explicitregistration by the user.

Also, in accordance with an embodiment of the present invention, theemotion-adaptive UI may operate on the basis of the emotion-adaptive UIpurpose 230.

FIG. 4 is a diagram illustrating a functional block structure of theemotion-adaptive UI performed in a wearable device or a portable deviceaccording to the present invention.

The present invention includes an emotion recognition unit 310, anemotion-adaptive UI execution unit 320, and an emotion-adaptive UI/UXmanagement unit 330.

A bio-signal recognition block 311 of the emotion recognition unit 310may collect bio-signals of the user 100 and process the bio-signals intoinformation for emotion recognition.

A voice/video recognition block 312 of the emotion recognition unit 310may extract information for emotion recognition from the voice/videodata of the user.

An emotion recognition model management block 313 may recognize a user'semotion state on the basis of the emotion recognition model and theextracted emotion recognition information.

The user-wearable device 110, the user portable device 120, and the IoTperipheral device 130 may exchange the user experience data and therecognized emotion information with each other. Herein, an inter-deviceinformation exchange management block 314 may be included for thepurpose of information exchange.

The user emotion information recognized from the emotion recognitionunit 310 may be transmitted to the emotion-adaptive UI execution unit320. Herein, the emotion-adaptive UI execution unit 320 is provided sothat the registered emotion-adaptive UI changes the pattern attributeaccording to the designated purpose. The emotion-adaptive UI executionunit 320 may include a UI control unit 321 and a UI execution unit 322.

The UI control unit 321 receives user emotion information from theemotion recognition unit 310. Also, the UI control unit 321 may receivethe end interface information for the emotion-adaptive UI from the user.In addition, the UI control unit 321 may perform interworking with theemotion-adaptive UI/UX management unit 330.

That is, the UI control unit 321 may determine the start and end pointsof the emotion-adaptive UI on the basis of the input user emotioninformation and end interface information, and the informationtransmitted by the emotion-adaptive UI/UX information management unit.In addition, the UI control unit 321 is provided so that the detailedpattern attributes of the corresponding emotion-adaptive UI iscontrolled to cause the detailed pattern attributes to be executed.

In this case, when adjusting the detailed pattern attribute of theemotion-adaptive UI, the detailed pattern attribute may be adjusted onthe basis of the usefulness information and the information provided inthe user experience information descriptor (UI-UX descriptor). Herein,the usefulness information corresponds to the information calculated onthe basis of the emotion state recognition information which varies inthe process of providing the service. Also, the user experienceinformation descriptor (UI-UX descriptor) is information includingusefulness information in the past, and corresponds to information onthe corresponding UI managed by the emotion-adaptive UI-UX informationmanager 332.

The emotion-adaptive UI-UX management unit 330 may include anemotion-adaptive UI information manager 331 and an emotion-adaptiveUI-UX information manager 332.

The emotion-adaptive UI information manager 331 may manage the attributeinformation of the registered emotion-adaptive UI.

The emotion-adaptive UI-UX information manager 332 may manage the userexperience information descriptor including the usefulness informationon operations of the emotion-adaptive UI.

In this case, the UI attribute information, the recognized emotioninformation, the detailed UI pattern attributes performed in therecognized emotion state, and the usefulness measurement values of thecorresponding patterns are stored as the user experience information ofthe corresponding emotion-adaptive UI in the user experience informationUX descriptor.

As an example of the emotion-adaptive UI operation of the presentinvention, the initial pattern attribute of the emotion-adaptive UI maybe determined and provided. According to an embodiment of the presentinvention, the information provided in the user experience informationdescriptor for the UI that was operated in the past may be utilized.That is, when the user experience information descriptor shows thecorresponding emotion information on the basis of the user emotioninformation recognized when a specific UI is operated, the UI patternattribute that showed the high usefulness in the past is determined sothat the initial emotion-adaptive UI may be performed with thecorresponding pattern. Herein, the experience information descriptor isused to calculate and store the usefulness information reflectingwhether the emotion information recognized at the time of start and endof the detailed pattern of the emotion-adaptive UI matches the purposeof the emotion-adaptive UI.

The emotion-adaptive UI of the present invention adapts to the user'semotion information based on the bio-signals and voice information ofthe user and thus changes the attribute of the detailed pattern of thecorresponding UI. To do this, when the corresponding emotion-adaptive UIis operated, the usefulness of the UI and the usefulness of the detailedpatterns constituting the UI are calculated and managed as userexperience information. The detailed pattern of the UI is adjusted withreference to the experience information stored in the past.

FIG. 5 is a diagram illustrating operations of an emotion-adaptive UIpattern according to an embodiment of the present invention.

FIG. 5 illustrates an operation for extracting the usefulness for aspecific k-th emotion-adaptive UI F_(k) in the emotion-adaptive UI-UXinformation management block and the usefulness information of the i-thdetailed pattern P_(k) _(i) 420 of the corresponding UI F_(k).

Emotion information S_(k) _(i) 430 recognized at the time of start andend of the i-th detailed pattern P_(k) _(i) of the UI(F_(k)), emotioninformation 440 and 450 recognized at the start point in time start_(F)_(k) 410 and end point in time End_(F) _(k) 420 of the corresponding UIF_(k), and the duration time duration_(K) _(s) 466 from the start untilthe end of the corresponding UI F_(k) are schematized.

Equation 1 shows how to calculate the usefulness Usefulness_(F) _(k) ofa particular emotion-adaptive UI F_(k). The usefulness Usefulness_(F)_(k) of a particular emotion-adaptive UI F_(k) of the present inventionis determined by utilizing a variation variation_(F) _(k) of useremotion information recognized at the time of start and end of thecorresponding emotion-adaptive UI and an effect effect_(F) _(k) onwhether the operation is performed in accordance with the purpose whichthe user wants to achieve through the corresponding emotion-adaptive UI.

In addition, when calculating the usefulness of the emotion-adaptive UI,it is possible to utilize information on a total of operation timeduration_(F) _(k) from the start until the end of the corresponding UI.

The effect information effect_(F) _(k) and the operation timeinformation duration_(F) _(k) used in calculating the usefulness of thecorresponding emotion-adaptive UI may be determined using weight valuesω_(F) _(k) ^(e) and ω_(F) _(k) ^(t) indicating the importance of eachinformation according to the emotion-adaptive UI purpose. Herein, theweight values ω_(F) _(k) ^(e) and ω_(F) _(k) ^(t) may be explicitlyspecified according to the system implementation method according to thepurpose of the emotion-adaptive UI, or may be designated as a result oflearning according to the implemented service and operation.

The computed usefulness of the UI may be stored in the user experienceinformation descriptor (UI-UX descriptor) and utilized in the serviceprocess. Equation 1 corresponds to an expression for calculating theusefulness of a specific UI F_(k).

Usefulness_(F) _(k) =ω_(F) _(k) ^(e)*effect_(F) _(k) +ω_(F) _(k)^(t)*duration_(F) _(k)

variation_(F) _(k) =sensing variation between S_(k) _(s) and S_(k) _(e)

effect_(F) _(k) =effect of variation_(F) _(k)   [Equation 1]

Equation 2 corresponds to an expression for calculating the usefulnessof a detailed pattern P_(k) _(i) belonging to a specific UI F_(k).

In the present invention, the usefulness Usefulness_(P) _(ki) of thedetailed pattern P_(k) _(i) belonging to a specific UI F_(k) isdetermined on the basis of the variation variation_(F) _(k) of useremotion information recognized on the basis of the user biometric signaland voice/video data measured at the start and end of theemotion-adaptive UI pattern and the effect effect_(F) _(k) on whetherthe operation is performed in accordance with the purpose which the userwants to achieve through the corresponding emotion-adaptive UI.

Usefulness_(P) _(k) =effect_(P) _(ki)

variation_(P) _(ki) =sensing variation between S_(k) _(i−1) and S_(k)_(i)

effect_(P) _(ki) =effect of variation_(P) _(ki)   [Equation 2]

FIG. 6 is a diagram illustrating calculation of usefulness informationon emotion variance to calculate the usefulness of an emotion-adaptiveUI and detailed pattern.

FIG. 6 is a diagram illustrating a method of calculating the effectinformation effect_(F) _(k) and effect_(P) _(ki) on whether theemotion-adaptive UI and the detail pattern conform to the purpose of thecorresponding UI according to an embodiment of the present invention

As shown in FIG. 6, it may be calculated by using the varianceinformation 470 between the purpose vector 490 and the operation vector480.

In this case, the operation vector may include the variance variance_(P)_(ki) of the emotion information S_(K) _(i−1) and S_(K) _(i) 430 thatare recognized during the operation.

Herein, the purpose vector may have a desirable degree of emotionvariance corresponding to the purpose 230 of the correspondingemotion-adaptive UI.

The purpose vector and the operation vector may be dynamically generatedon the basis of the emotion information measured at the start and endpoints of the interval in which the usefulness is to be measured withinthe emotion-adaptive UI operation interval.

As an example of the usefulness calculation of the present invention, adifference between the purpose vector and the operation vector iscalculated by applying a method such as Euclidean distance calculation,cosine similarity, and the like, in which the closer the distance is tothe purpose vector, the higher the usefulness is calculated. The desiredvariance direction and variance amount of the purpose vector of theemotion-adaptive UI may be designated explicitly by the service and theuser depending on the system implementation method or by the learningresult using data such as the user experience data.

FIG. 7 is a block diagram illustrating an emotion-adaptive UI executionunit 320 according to the present invention.

The emotion-adaptive UI execution unit 320 may include a UI controller321 and a UI executor 322. The UI controller may include functionmodules for display control, vibration control, sound control, andtactile control of a user wearable device and a portable interfacedevice.

FIG. 8 is a diagram illustrating an embodiment of a wrist wearabledevice 110 that supports executing the emotion-adaptive UI according tothe present invention.

A wearable wrist device according to the present invention may include abio-signal recognition module 311, and emotion-adaptive UI control andexecution modules 321 and 322 such as display and vibration modules.

The service of the present invention may be configured with at least oneof a service operation condition, a service end condition, and anemotion-adaptive UI service type on the basis of the purpose informationof the service. Therefore, the user may register the configurations, orthe above configurations may be automatically set by the providedservice.

More specifically, time, sensitivity, emotion-adaptive UI purpose, andemotion-adaptive UI interface type for the emotion-adaptive UI operationstate according to the present invention may be explicitly set through auser's registration through a wearable device or automatically set withinterworking with driven services (210-240).

In addition, in the case of an apparatus providing a tactile type, theUI interface may include the module 322 for providing a tactile type UIof a hardware device 322-1 for providing tactile in up, down, left,right, and composite direction.

FIG. 9 is an operation flowchart illustrating an emotion-adaptive UIaccording to an embodiment of the present invention.

The user may register an emotion-adaptive UI of a specificemotion-adaptive UI operation state through a combination of conditionsfor time and emotion attributes of a specific emotion-adaptive UIoperation state in which the emotion-adaptive UI operates (S901).

The system of the present invention may manage the attribute informationof the emotion-adaptive UI registered by the user (S902).

The user emotion information may be recognized on the basis of thebio-signal and voice/video data of the user (S903).

The state recognition may be performed on whether the emotion-adaptiveUI should be executed through the user emotion information and thecontext information such as time designated in the emotion-adaptive UIoperation state (S904).

For the emotion-adaptive UI operation, the UI-UX descriptor informationhaving the purpose of the emotion-adaptive UI may be referenced (S905).

In this case, it is possible to utilize the emotion-adaptive UI-UXinformation descriptor, in which the usefulness information on the UIpattern is managed for the purpose of the emotion service. Theemotion-adaptive UI may be executed by referring to the detailed patterninformation (S906) on the basis of the operation information of the UI,which shows the high usefulness in association with the emotion-adaptiveUI-UX information descriptor, the recognized user biorhythm, and theemotion information (S907).

The usefulness information may be calculated in accordance with theemotion information extracted on the basis of the bio-signal andvoice/video data of the user and the purpose type of the correspondingUI specified by the user (S908). The usefulness information of theemotion-adaptive UI-UX may be updated on the basis of the calculatedusefulness information (S909).

The type of the pattern to be executed next may be determined using thecalculated usefulness information of the detailed pattern and theinformation stored in the UI-UX descriptor (S906).

Further, in adjusting the detailed pattern of the emotion-adaptive UI,it is determined whether the usefulness value is equal to or greaterthan a threshold value. In this case, when the usefulness calculated forthe pattern that was previously executed is equal to or higher than athreshold value and thus is operated in accordance with the purpose ofthe emotion-adaptive UI, the corresponding pattern may be maintained tobe executed (S907).

The usefulness information Usefulness_(P) _(i) may be stored in theemotion-adaptive UI-UX descriptor (S909). In this case, the usefulnessinformation Usefulness_(P) _(i) may be calculated by reflecting whetherthe emotion information recognized at the time of start and end of thedetailed pattern of the emotion-adaptive UI conforms to the purpose ofthe emotion-adaptive UI.

The end of the emotion-adaptive UI may be recognized in accordance withan explicit user expression as a specific word such as “stop”, anexplicit interface such as a button, the time designated in thecorresponding emotion-adaptive UI operation state, and a case where theuser emotion is changed to the purpose emotion designated in thecorresponding UI (S910).

When it is recognized that the emotion-adaptive UI is terminated, theemotion-adaptive UI-UX information management function block extractsand updates the usefulness information for the correspondingemotion-adaptive UI (S911).

The present invention relates to a method of providing anemotion-adaptive UI, including a method in which an operation purposefor an emotion-adaptive UI is designated from the user and the service,a method of measuring the usefulness of the emotion-adaptive UI and thedetailed pattern on the basis of the purpose information of theemotion-adaptive UI at the process of executing the emotion-adaptive UI,a method of storing the user experience information on the usefulness ofthe detailed pattern at the process of executing the emotion-adaptiveUI, and a method of adjusting the detailed pattern of the correspondingUI on the basis of the usefulness of the detailed pattern measured atthe process of executing the emotion-adaptive UI and the user experienceinformation on the usefulness of each pattern stored.

Further, according to an embodiment of the present invention, a wearabledevice with an emotion-adaptive UI can be provided. Herein, according toan embodiment of the present invention, a wearable wrist device can beprovided, the device including a collection unit for collectingbio-signals for recognizing user's biorhythm and emotion, andexperiential data such as voice, and performing a pattern change intactile and intensity in up, down, left, right, and composite directionin order to support emotion-adaptive UI of tactile type.

1. A method of providing an emotion-adaptive user interface (UI) on thebasis of affective computing, the method comprising: acquiring an inputsignal for emotion information of a user; providing service based on theemotion information through the emotion-adaptive UI; and calculatingusefulness information for the service; and when the service isprovided, detailed pattern of the emotion-adaptive UI is changed andprovided on the basis of at least one of purpose information and theusefulness information included in user experience information.
 2. Themethod of claim 1, wherein the detailed pattern usefulness informationof the usefulness information is calculated on the basis of at least oneof a difference between the emotion information at the start of thedetailed pattern and the emotion information at the end of the detailedpattern and the purpose information of the service.
 3. The method ofclaim 2, wherein the emotion information is generated based on at leastone of bio-signal data, voice data, and video data of the user.
 4. Themethod of claim 1, wherein the usefulness information corresponds to anevaluation index for the service provided through the emotion-adaptiveUI, and is determined on the basis of the purpose information of theservice.
 5. The method of claim 1, wherein when the service is provided,the detailed patterns included in the service are changed in a manner tosatisfy the purpose information of the service on the basis of thepurpose information of the service.
 6. The method of claim 3, wherein ina case that the detailed pattern is changed in a manner to satisfy thepurpose information of the service, when the usefulness informationcalculated for the detailed pattern is equal to or greater than athreshold value according to a result of providing the service havingthe detailed pattern, it is determined that the purpose information ofthe service is satisfied.
 7. The method of claim 1, wherein the serviceusefulness information of the usefulness information is calculated onthe basis of at least one of the emotion information at the start of thedetailed pattern of the service, the emotion information at the end ofthe detailed pattern, and the purpose information of the service.
 8. Themethod of claim 1, wherein the service is configured with at least oneof a service operation condition, a service end condition, and anemotion-adaptive UI service type on the basis of purpose information ofthe service, and wherein the emotion-adaptive UI service type isconfigured with one or more detailed patterns.
 9. The method of claim 8,wherein the service operation condition and the service end conditioninclude at least one of time information for operating the service,current emotion information, target emotion information, and placeinformation.
 10. The method of claim 8, wherein the emotion-adaptive UIservice type includes a sound service, a display service, a vibrationservice, and a tactile service, and the service provided through theemotion-adaptive UI includes at least one of a sound service, a displayservice, a vibration service, and a tactile service.
 11. The method ofclaim 8, wherein the detailed pattern corresponds to the detailedattribute for the emotion-adaptive UI service type, and corresponds to amethod of controlling the emotion-adaptive UI service type.
 12. Themethod of claim 1, wherein when the service is provided, the purposeinformation of the user experience information for the service ischecked, and the detailed pattern included in the service is determinedon the basis of the usefulness information included in the correspondinginformation.
 13. The method of claim 1, wherein the input signal for theuser emotion information includes at least one of bio-signal data, voicedata, and video data of the user, and the emotion information isrecognized on the basis of the input signal using an emotion recognitionmodel.
 14. The method of claim 1, wherein when default purposeinformation is selected as the purpose information of the service, thedefault purpose information includes emotion awakening purpose, emotioncontrol purpose, and emotion consolation purpose.
 15. The method ofclaim 1, wherein when a feedback is reflected on the operation of theemotion-adaptive UI based on the usefulness information for the providedservice, information on the detailed pattern for the provided service isupdated.
 16. An apparatus for providing an emotion-adaptive userinterface (UI) on the basis of affective computing, the apparatuscomprising: an emotion recognition unit acquiring an input signal foremotion information of a user; an emotion-adaptive UI execution unitproviding a service based on the acquired input signal for the emotioninformation through the emotion-adaptive UI and calculating usefulnessinformation for the service; and an emotion-adaptive UI/UX managementunit reflecting a feedback based on the usefulness information for theprovided service, wherein when the service is provided, the detailedpattern is changed and provided on the basis of at least one of thepurpose information and the usefulness information included in userexperience information.
 17. The apparatus of claim 16, wherein thedetailed pattern usefulness information of the usefulness information iscalculated on the basis of at least one of a difference between theemotion information at the start of the detailed pattern and the emotioninformation at the end of the detailed pattern and the purposeinformation of the service.
 18. The apparatus of claim 16, wherein theapparatus is provided to operate one or more emotion-adaptive UI servicetypes from sound service, display service, vibration service, andtactile service.
 19. The apparatus of claim 16, wherein the apparatuscorresponds to a wearable device.
 20. The apparatus of claim 16, whereinwhen the service is provided, the detailed patterns included in theservice are changed in a manner to satisfy the purpose information ofthe service on the basis of the purpose information of the service.